Reestablishing redundancy in redundant storage

ABSTRACT

Storage redundancy may be resynchronized without determining a snapshot difference. A storage component ( 210 ) owning a volume ( 122 ) can maintain current and expected generation numbers ( 212, 214 ) based on modification requests received and modification requests that a backup component ( 220 ) acknowledges completing. The backup ( 220 ) can maintain current and expected generation numbers ( 222, 224 ) based on modification requests received and applied to a backup volume ( 124 ). If either component ( 210, 220 ) fails and later returns to service, differences between the owner&#39;s current and expected generation numbers ( 212, 214 ) and the backup&#39;s current and expected generation numbers ( 222, 224 ) indicate which modification requests may have been missed and need to be reconstructed to restore synchronization.

BACKGROUND

Many storage systems provide redundancy to cope with failures. Suchredundancy is often achieved by replicating stored data, i.e., storingthe same data in multiple locations. Data in a storage system aretypically organized in volumes, e.g., virtual volumes, with each volumebeing known to client systems, so that client systems logically readdata from and write data to the volumes. Storage systems track wheredata for the virtual volumes is physically stored, and storage systemsproviding redundancy for fault tolerance purposes replicate the volumedata in different physical storage. One way for a storage system toreplicate volume data uses a primary storage system and a backup storagesystem. In such a system, clients normally issue IO requests for a givenvolume to the primary storage system, but if the primary storage failsor is unavailable, clients may instead issue such IO requests to thebackup storage system. In such a scenario, the primary storage systemmay be responsible for updating the backup storage system with any datathat the primary system receives for writing to a given volume.

In general, either the primary or the backup storage system may fail atany time. If the primary storage system fails, the backup storage systemmay become the primary storage system and take over IO responsibilitiesfor volumes that the failed storage had provided. If the backup fails,the primary may continue to be responsible for IO for the volumes butwithout concurrently backing up data. In either failure scenario, astorage system may operate without redundancy until the failed systemresumes normal function. When the failed system returns to service, thereturning system may assume a backup role but may have missed datachanges that the primary storage system completed while the returningsystem was unavailable. In this case, the backup system then needs toupdate the backup data. Once the backup storage system for a volume hasdata identical to the data the primary storage system has for thevolume, full redundancy has been reestablished, and both the primary andbackup storage systems can replicate new data changes.

Some traditional solutions for reestablishing redundancy involve takingsnapshots of the volume on the primary system after the failure of theredundant storage system and determining a difference between thatsnapshot and a later snapshot taken when the redundant storage systemreturns to service. The difference indicates which data the backupstorage system needs to backup to reestablish redundancy. Determiningdata differences between snapshots can be time consuming particularlywhen the storage capacity of a volume is large. Efficient systems andmethods are needed for reestablishing redundancy for replicated storagevolumes after a storage processor failure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an architecture including astorage platform in accordance with one example of the presentdisclosure.

FIG. 2 is a block diagram illustrating a storage platform including aprimary storage component and a backup storage component in accordancewith one example of the present disclosure.

FIG. 3-1 is a flow diagram of a process that a primary storage componentin accordance with an example of the present disclosure performs uponreceiving a modification request.

FIG. 3-2 is a flow diagram of a process that a backup storage componentin accordance with an example of the present disclosure performs uponreceiving a modification request forwarded from the primary storagecomponent.

FIG. 3-3 is a flow diagram of a process that a primary storage componentin accordance with an example of the present disclosure performs uponreceiving an acknowledgement from the backup storage component.

FIGS. 4-1, 4-2, and 4-3 are block diagrams illustrating states of astorage platform when processing an example modification request.

FIG. 5 is block diagrams illustrating a state of the storage platformwhen processing a series of modification requests.

FIG. 6 is a flows diagram of a process for primary and backup storagecomponents to reestablish redundancy.

The drawings illustrate examples for the purpose of explanation and arenot of the invention itself. Use of the same reference symbols indifferent figures indicates similar or identical items.

DETAILED DESCRIPTION

Storage systems and methods in accordance with specific examples of thepresent disclosure may resynchronize redundant storage, i.e.,reestablish redundancy, without taking snapshots or calculating asnapshot difference. Such systems and methods may particularly use asequence of generation numbers that uniquely identify modificationrequests and indicate the order of modification requests. A primarystorage component that owns a volume can record resynchronization dataincluding a set of generation numbers to identify which modificationrequests a backup storage component has acknowledged as being backed up,and the backup storage system can similarly record resynchronizationdata including generation numbers that identify which modificationrequests the backup storage component has processed. If either storagecomponent fails and subsequently returns, the storage components can usethe resynchronization data to determine which modification requests areneeded to resynchronize a backup volume with a primary volume, and theprimary storage component can reconstruct and send the need modificationrequests to the backup storage component.

In one specific example, a storage platform includes multiple storagecomponents and assigns volumes of storage to the storage components. Thestorage component to which a volume is assigned is sometimes referred toas the owner of the volume or as the primary storage component for thevolume, and the storage component that owns a volume normally performsIO requests, e.g., writes and reads, associated with the volume. Thestorage platform may also assign another of the storage components as abackup storage component that is responsible for retaining a copy ofdata of the volume. During normal operation of the storage platform, allmodification requests associated with a volume or a storage location inthe volume, e.g., a volume ID and an offset, are sent to the primarystorage component for the volume, which associates or assigns respectivegeneration numbers from a monotonically increasing sequence ofgeneration numbers respectively with the modification request. Themodification request are generally IO requests of specific types (e.g.,write, unmap, xcopy, write-same, and createsnapshot instructions) thatmodify stored data. As a result, each IO request that changes data in avolume may be uniquely identified and distinguished by the generationnumber of the IO request. The primary storage component and the backupstorage component use generation numbers in databases for storagefunctions such as tracking where data corresponding to volumes arephysically stored. The primary and backup storage components also usethe generation numbers to record which IO requests were stored in theprimary and backup volumes.

In accordance with a further aspect of the current disclosure, theprimary storage component and the backup storage component for a volumemay independently record two generation number values, sometimesreferred to as the current generation number and the expected generationnumber. When the primary storage component and the backup storagecomponent are synchronized, the current generation number and expectedgeneration number of the primary storage component respectively matchthe current generation number and expected generation number of thebackup storage component. During operation, however, the values ofcurrent generation number and the expected generation number in theprimary storage component and the backup storage component may differ,for example, when the backup storage component has not completed backupof all of IO requests that the primary storage component has completed.When the primary or backup storage component fails or otherwise becomesunavailable, the functioning storage component acts as the owner of thevolume, and the current generation number in the available storagecomponent may become increasingly different from the current generationnumber in the unavailable storage component. When the storage componentlater returns, persistent values of the current and expected generationnumbers in the returned storage component may be used in a process thatresynchronizes backup storage data for the volume.

FIG. 1 is a block diagram including a storage platform 100 in accordancewith one example of the present disclosure. Storage platform 100includes one or more host servers 110-1 to 110-n, which are genericallyreferred to herein as host server(s) 110. Each host server 110 may be aconventional computer or other computing system including a centralprocessing unit (CPU), memory, and interfaces for connections tointernal or external devices. One or more storage processing units(SPUs) 120-1 to 120-x, which are generically referred to herein asSPU(s) 120, are installed in host servers 110. In general, storageplatform 100 may include one or more host server 110. For redundancy,storage platform 100 includes at least two host servers 110 or at leastat least two storage processing units 120, but more generally, alimitless number of different configurations are possible containing anynumber of host servers 110 and any number of SPUs 120.

Each SPU 120 may provide storage services to the host servers 110 orclients 162 via virtual volumes. FIG. 1 particularly shows SPU 120-1provides storage services relating to a set of virtual volumes 122 a to122 b and shows SPU 120-x provides storage services relating to virtualvolumes 122 y to 122 z. SPU 120-1 is sometimes referred to as “owning”volumes 122 a to 122 b in that SPU 120-1 is normally responsible forfulfilling IO requests that are directed at any of volumes 122 a to 122b. Similarly, SPU 120-x owns volumes 122 y to 122 z in that SPU 120-x isnormally responsible for executing IO requests that are directed at anyof volumes 122 y to 122 z. The virtual volumes, e.g., volumes 122 a to122 b and 122 y to 122 z, in FIG. 1 are generically referred to hereinas volume(s) 122 or primary volume(s) 122. Each SPU 120 controlsassigned backend storage for local storage data corresponding to volumes122 that the SPU 120 owns. Additionally, each SPU 120 may maintain a setof one or more backup volumes that correspond to virtual volumes 122owned by one or more other SPUs 120. In the example of FIG. 1 , SPU120-1 maintains backup volumes 124 y to 124 z that are intended to copyvirtual volumes 122 y to 122 z that SPU 120-x owns, and SPU 120-xmaintains backup volumes 124 a to 124 b that are intended to copyvirtual volumes 122 a to 122 b that SPU 120-1 owns. One or more backupvolumes 124 a to 124 b and 124 y to 124 z of FIG. 1 are genericallyreferred to herein as backup volume(s) 124. As described further below,backup volumes 124 may be virtual volumes that are generally copies ofrespective virtual volumes 122 but may be incomplete copies to theextent that the owner SPU 120 may have completed one or more IO requeststhat the backup SPU 120 has not yet completed.

Each SPU 120 controls or uses its physical storage 150, e.g., storage150-1 and 150-x respectively for SPUs 120-1 and 150-x in FIG. 1 , tostore the data associated with virtual volumes 122 and backup volumes124. Storage 150 may be installed devices in the same host server 110 asan associated SPU 120, may include one or more external storage devicesdirectly connected to its associate SPU 120, or network-accessiblestorage devices 150-n. Storage 150 may employ, for example, hard diskdrives, solid state drives, or other nonvolatile storage devices ormedia in which data may be physically stored, and storage 150particularly may have a redundant array of independent disks (RAID) 5 or6 configuration for performance and redundancy.

Each SPU 120 includes memory 130 and a processing system 140. Memory 130may provide general purpose memory for processes that processing system140 executes and may particularly store a data index 138 for trackingwhere data corresponding to its volumes 122 and 124 are physicallystored in its backend media 150. More specifically SPUs 120-1 to 120-xhave respective data indexes 138-1 to 138-x, generically referred to asdata indexes 138, to track where data is physically stored. Each dataindex 138 may, for example, be a key-value store in which each entry hasa key including a volume ID of a target virtual volume, an offset in thetarget virtual volume, and a generation number for an 10 request thatwrote to that offset in the target virtual volume. The value in eachentry indicates a physical location in storage 150 when the write dataof the IO request is stored.

Processing system 140 may include one or more processors or processingcores to execute software or firmware to implement an IO processor 142and a maintenance module 144. IO processor 142 may be used to process IOrequests such as reads and writes to virtual volumes 122 owed by the SPU120, and maintenance module 144 may perform operations such as garbagecollection to ensure that the associated physical storage 150, dataindex 138, and other resources of the SPU 120 are efficiently used andmaintained. More generally, SPUs 120 may provide storage services andfunctionality such as write and read (TO) operations, compression,deduplication, encryption, mirroring, remote and local access to data,disaster recovery and local and/or cloud backup. U.S. patent applicationSer. No. 16/748,454, entitled “Efficient IO Processing in a StorageSystem with Instant Snapshot, Xcopy, and Unmap Capabilities,” filed Jan.21, 2020, and U.S. patent application Ser. No. 16/783,035, entitled“Primary Storage with Deduplication,” filed Feb. 5, 2020 furtherdescribe some implementations of the data services that SPU's 120 mayprovide and are hereby incorporated by reference in their entirety.

Multiple SPUs 120, e.g., SPU 120-1 to 120-x in FIG. 1 , may be connectedtogether using high speed data links 166, e.g., one or more parallel 25GB/s Ethernet links, to form a data network for a set of associated SPUs120 sometimes referred to herein as a pod of SPUs. Each SPU 120 mayfurther employ a link 168 to connect to a local network 160 that is alsoconnected hosts 110 and client stations 162. Each client station 162 maybe a computer including a processor, memory, and software or firmwarefor executing a user interface adapted to communicate over local network160. Client stations 162 in general may be storage clients that usestorage services that storage platform 100 provides. FIG. 1 furthershows that local network 160 may provide a connection through a firewall164 to a public or wide area network 170, e.g., the Internet, so thatSPUs 120 may be remotely accessed, for example, by a cloud-basedmanagement infrastructure 180.

Storage platform 100, in one specific implementation of the presentdisclosure, provides an interface that exposes the virtual volumes 122to storage operations such as writing and reading of blocks or pages ofdata at virtual locations in volumes 122. For example, SPUs 120 maypresent storage volumes 122 to host servers 110 via SCSI (small computersystem interface) target, NVMe (nonvolatile memory express) virtualtarget, or other data interface. The locally installed storage 150 mayprovide the physical storage of data patterns of data written to virtualvolumes 122. As previously noted, storage 150 may include storagedevices, e.g., HDDs, SSDs or NVM, installed in the host server 110,connected through an interface (e.g., a RAID card) of the host server110, or connected directly to the SPU 120. In some implementations, eachstorage device providing physical storage 150 is only visible to thelocal SPU 120. In particular, the CPUs, OS, firmware, and baseboardmanagement controllers (BMCs) of host servers 110 are not able to see ordirectly access some or all of storage 150. Host servers 110 and clients162 may instead request data services on the virtual volumes 122 ownedby SPUs 120.

A read or write request may identify a storage location by a volumenumber (also referred to as a volume ID) of a virtual volume 122 and anoffset to a starting location of affected data in the identified virtualvolume 122. A client 162 may send an IO request directed at a virtualvolume 122 to a host 110 or directly to the SPU 120 that owns thevirtual volume 122. In some examples of the present disclosure, the SPU120 that owns the target volume 122 tags modification requests withgeneration numbers. In particular, an IO processor 142 in an SPU 120that receives and processes a modification request updates, e.g.,increments, a current generation number kept for the target volume 122and assigns the incremented value as the generation number of themodification request. As a result, different modification requests(whether to the same or different virtual locations in a volume 122) areeasily distinguishable. In an exemplary implementation, each SPU 120maintains in memory 130 a current generation number and an expectedgeneration number for each volume 122 that the SPU 120 owns. In theexample of FIG. 1 , SPU 120-1 records current generation numbers 131-1and expected generation numbers 132-1 for the virtual volumes 122 a to122 b in a persistent, non-volatile portion of its memory 130-1, and SPU120-x records current generation numbers 131-x and expected generationnumbers 132-x for the virtual volumes 122 y to 122 z in a persistent,non-volatile portion of its memory 130-x. Each current generation number131-1 or 131-x corresponds to the generation number that was assigned tothe last IO request changing data in the volume 122 associated with thegeneration number, and each expected generation number 132-1 or 132-xcorresponds to a generation number such that all IO requests withgeneration number lower than that number have been completed on both theprimary and backup and have been acknowledge to the client requestingthe 10. Typically, upon receiving a modification request to a targetvolume 122, the SPU 120 that owns the target volume 122 increments thecurrent generation numbers associated with the targeted volume andassigns the current generation number to the request, and the SPU 120owning the target volume may update the expected generation number forthe target volume in response to acknowledgments from the SPU 120 thatis the backup storage components of the target volume.

Each SPU 120 may also record in its memory 130 a current generationnumber 133 and an expected generation number for each backup volume 124that the SPU 120 maintains. In the example of FIG. 1 , SPU 120-1 recordscurrent generation numbers 133-1 and expected generation numbers 134-1for the backup volumes 124 y to 124 z, and SPU 120-x records currentgeneration numbers 133-x and expected generation numbers 134-x for thebackup volumes 124 a to 124 b. As described further below, an SPU 120receiving a forwarded IO request for a backup volume 124 updates itscurrent generation number 133 upon applying an IO request having ageneration number greater than the value of the current generationnumber 133 for the backup volume 124 and update its expected generationnumber 134 for a backup volume 124 upon applying an IO request having ageneration number equal to the expected generation number 134 for thebackup volume 124.

FIG. 2 illustrates a storage platform 200 in accordance with anotherexample of the present disclosure including a primary storage component210 that owns a virtual volume 122 and a backup storage component 212that maintains a backup volume 124 that mirrors volume 122. In general,each storage components 210 and 220 may include an SPU with backendstorage and may own or maintain multiple primary and/or backup volumesas described above with reference to FIG. 1 . FIG. 2 shows just virtualvolume 122 and backup volume 124 for ease of description.

Primary storage component 210 owns primary volume 122 and keeps metadataincluding a current generation number 212, an expected generation number214, a bitmap 216, and a data index 217 for primary volume 122. Backupstorage component 220 maintains backup volume 124 and keeps metadataincluding a current generation number 222, an expected generation number224, a bitmap 226, and a data index 227 for backup volume 124. Currentgeneration number 212 is the generation number of the last modificationrequest that primary storage component 210 started, and expectedgeneration number 214 is next generation number after the highestgeneration number that both primary storage component 210 and backupstorage component 220 acknowledged. More generally, storage component210 may maintain a sequence window, e.g., a current generation numberand an expected generation number, for each volume family, e.g., eachwritable volume and its read-only snapshots together, that storagecomponent 210 owns.

Backup storage component 220 similarly represents a sequence windowusing current generation number 222 and expected generation number 224.Current generation number 222 and expected generation number 224respectively hold the same values as current generation number 212 andexpected generation number 214 in primary component 220 when volumes 122and 124 are synchronized, but at other times, current generation number222 and expected generation number 224 may differ from currentgeneration number 212 and expected generation number 214 in primarystorage component 210 as described further below.

Each storage component 210 or 220, in the example of FIG. 2 , furtherincludes a bitmap 216 or 226 identifying the generation numbers of IOrequests that storage component 210 or 220 have completed, i.e.,recorded in persistent storage 251 or 252. (Persistent storage 251 or252 may include backend media and/or a non-volatile memory or buffer inwhich data is stored for subsequent writing to the backend media.)Bitmap 216 in primary storage component 210 generally tracks generationnumbers which primary storage component 210 knows have completed on boththe owner and backup storage component, while bitmap 226 in backupstorage component 220 tracks generation numbers that have completed onthe backup storage component 220. In particular, bits in bitmap 216 mayrespectively correspond to generation numbers in the sequence window ofprimary storage component 210, and each bit is set or not to indicatewhether data from the corresponding IO request has been saved inpersistent storage 251. Bits of bitmap 226 corresponds to generationnumbers and are set or not to indicate the generation numbers of IOrequests that backup storage component 220 applied, e.g., saved inpersistent storage 252 and acknowledged to primary storage component210. In some implementations, the information provided by bitmap 216 or226 may be derived from data index 217 or 227. In some otherimplementations, data may first be saved to persistent storage 251 or252 and acknowledged before a block from persistent storage 251 or 252,e.g., data corresponding to several write operations, is stored inbackend storage. In which case, a combination of persistent storage 251or 252 and data index 217 or 227 can be used to derive bitmap 216 or226. In either case, bitmaps 216 and 229 may not strictly be required,but use of bitmaps 216 and 217 as described further below may providefaster operations of storage components 210 and 220.

Data indexes 217 and 227 indicate where data of primary volume 122 andbackup volume 124 are stored in physical storage. In particular, dataindex 217 or 227 may be a database of key-value entries. In each entry,the key represents a generation number of an IO request and a virtualvolume location where the IO request wrote data, and the valuerepresents where the data is physically stored. Reading the datacorresponding to a virtual volume location (or to an IO request) maythus be completed by finding the entry in data index 217 having a keycorresponding to the last write (or to the last modification request)targeted at the virtual volume location and then reading from thephysical location indicated in the data index entry found. As describedfurther below, a resynchronization process may use the data index 217 toreconstruct IO requests that may be needed for resynchronization.

FIG. 3-1 is a flow diagram of a process 310 that a primary storagecomponent may employ when processing an IO request, e.g., a writerequest from a storage client, that modifies data in a virtual volumethat the primary storage component owns and that is mirrored in a backupstorage component. To provide a specific example, process 310 isdescribed with reference to storage platform 200 of FIG. 2 where primaryvolume 122 owned by storage component 210 is mirrored by backup volume124, which storage component 220 maintains. Method 310 may, of course,be employed in the same or similar manner in other storage platforms.During normal operation of storage platform 200, all IO requestsassociated with volume 122 or a storage location, e.g., a volume ID andan offset, in volume 122 are sent to primary storage component 210, andprimary storage component 210 in block 311 receives an IO request tomodify the content of owned volume 122.

Primary storage component 210, in process block 312, assigns to the IOrequest a unique generation number from a monotonically increasingsequence of generation numbers, e.g., an ordered sequence such as thesequence of consecutive integers. In particular, primary storagecomponent 210 in performing process block 312 may increment the value ofcurrent generation number 212 and assign the incremented value to thenewly received IO request. In a process block 313, primary storagecomponent 210 updates its current generation number 212, e.g.,increments the current generation number for the volume family that theIO request targets. Primary storage component 210, in a process block314, applies the IO request, e.g., stores data locally in persistentstorage 251 to be flushed eventually to backend storage and updates dataindex 217, and then, in a process block 315, reports completion of theIO operation, e.g., completion is reported locally so that bitmap 216 isupdated. (When backup 220 also reports completion, primary storagecomponent 210 may report completion to the storage client.) In a processblock 316, the primary storage component 210 sends a copy of the IOrequest with the assigned generation number to backup storage component220.

In response to a series IO requests modifying volume 122, primarystorage component 210 may repeatedly perform process 310 and incrementcurrent generation number 212 for the owned volume 122 and repeatedlyforward the IO requests to the backup storage component 220. Sinceprimary and backup storage components 210 and 220 generally operateasynchronously and since IO requests may be communicated through anetwork, backup storage component 220 may receive IO requests fromprimary storage component 210 in any order. Accordingly, backup storagecomponent 220 may also process IO requests from primary storagecomponent 210 in any order.

FIG. 3-2 is a flow diagram of a process 320 that backup storagecomponent 220 of FIG. 2 may employ when processing a forwarded IOrequest, e.g., a write request, that changed data in the virtual volume122 owned by primary storage component 210. In a process block 321,backup storage component 220 receives the IO request with the generationnumber from primary storage component 210, and backup storage component220, in a process block 322, updates its current generation number 222for backup volume 124 if the generation number of the IO request islarger than any previously-seen generation number for volume 124. Backupstorage component 220 in a process block 323 applies the requested IO onbackup volume 124, e.g., persists the data in persistent storage 252 andupdates bitmap 226. To update bitmap 226, backup storage component 220may set the bit in bitmap 226 corresponding the generation number of theIO request to indicate that the IO request was applied. When the data iswritten to backend media, backup storage component 220 updates dataindex 227, e.g., creates an entry in data index 227 mapping thegeneration number and the target location in virtual volume 124 to aphysical location of the written data in the backend media. Backupstorage component 220, in a process block 324, send an acknowledgementof the IO request to primary storage component 210. Backup storagecomponent 220 in a process block 325 updates its expected generationnumber 224 for volume 124 if the acknowledged IO request has ageneration number matching expected generation number 224 of backupvolume 124. The update in process block 325 increases generation number224 to the next generation number that has not been applied. The updatemay, for example, scan bitmap 226 to find the first unset bit and thenset expected generation number 224 to the generation numbercorresponding to that bit.

The backup storage component 220 generally sends acknowledgements of IOrequests to primary storage component 210 as backup storage component220 completes backup of the IO requests. As a result, primary storagecomponent 210 may receive acknowledgements from backup storage component220 in any order. FIG. 3-3 is a flow diagram of a process 330 thatprimary storage 210 may perform when receiving an acknowledgement.Primary storage component 210 receives the acknowledgement in a processblock 331 and then, in block 332, updates bitmap 216, e.g., sets a bitcorresponding to the acknowledged generation number in bitmap 216. In aprocess block 333, primary storage component 210 updates expectedgeneration number 214, if the IO request with the expected generationnumber was acknowledged. This update may based on first unset bit inbitmap 216. Alternatively, the update of expected generation number 214more specifically sets (or keeps) expected generation number 314 at avalue equal to the generation number of the first not yet acknowledgedIO request. In general, expected generation 214 is always the lowestunseen generation number, so if all IO requests have completed, expectedgeneration number 214 is updated to one greater than the highestacknowledged IO request.

FIGS. 4-1, 4-2, and 4-3 illustrate states of storage platform 200 ofFIG. 2 when processing a modification request requiring backup. FIG. 4-1shows the state of storage platform 200 when primary storage component210 receives a modification request 410 directed to an owned volume.Initially, a backup of the volume is synchronized, and in primarystorage component 210 and backup storage component 220, their currentgeneration numbers 212 and 222 for the volume family have a value G0,and their expected generation numbers 214 and 224 for the volume familyhave a value G0+1. In response to the IO request, primary storagecomponent 210 assigns generation number G0+1 to the IO request, appliesthe modification locally, e.g., commits data from the IO request tonon-volatile storage 251, and sends the IO request with generationnumber G0+1 to backup storage component 220 as shown in FIG. 4-2 . Ingeneral, committing data to persistent storage 251 on primary storagecomponent 210 and sending the IO request to backup storage component 220may happen in parallel, and not sequentially. At this time, primarystorage component 210 also sets its current generation number 212 toG0+1.

Backup storage component 220, upon receiving the IO request withgeneration number G0+1, applies the IO request e.g., persists the IOdata in non-volatile storage 252, sets current generation number 222 tovalue G0+1, updates bitmap 226 by setting the bit that corresponds togeneration number G0+1, and sends an acknowledgement of IO request G0+1back to primary storage component 210 as shown in FIG. 4-3 . Since thegeneration number G0+1 of the acknowledged IO request was equal to thevalue of expected generation number 224, backup storage component 220also updates expected generation number 224 to G0+2. Primary storagecomponent 210 after applying IO request G0+1 locally and receiving theacknowledgement of IO request G0+1 from backup storage component 220,sets expected generation number 214 to G0+2 and updates bitmap 216 bysetting the bit that corresponds to generation number G0+1. At thattime, the primary and backup volumes are again synchronized, and bothprimary storage component 210 and backup storage component 220 havecurrent/expected generation numbers set to G0+1/G0+2 for the volumefamily.

A primary storage component may receive multiple IO requests at the sametime (or in a short period of time) targeted at the same owned volume,resulting the primary storage component forwarding one or more IOrequests to a backup storage component before receiving acknowledgementsof prior IO requests. These IO requests may be sent to the backupstorage component in parallel or sequentially, but network transmission(or other factors) may cause the copied IO requests to be seen andapplied by the backup storage component in any order. Similarly, theprimary storage component can receive and process acknowledgements fromthe backup storage component in any order.

FIG. 5 illustrates a state of storage platform 200 after primary storagesystem 210 received five IO requests that change the content of ownedvolume 122. In the illustrated example, primary storage component 210and backup storage component 220 started in sync with current generationnumbers 212 and 222 having value G0 and expected generation numbers 214and 224 having value G0+1. Owner 210 upon receiving the five IO requestsincrements current generation number 212 five times to G0+5 and sendsfive IO requests corresponding to generation numbers G0+1, G0+2, G0+3,G0+4, and G0+5 to backup storage component 220. In the illustratedexample, primary storage component 210 locally completes G0+1, G0+2, andG0+3. In the illustrated example, backup storage component 220 saw andapplied the IO requests corresponding to generation numbers G0+2 andG0+4, but owner 210, so far in FIG. 5 , has only seen theacknowledgement of IO request G0+2 from backup storage component 220.Acknowledgement for G0+4 may be in flight from backup storage component220 to primary storage component 210 or may not yet have been sent.

In the state of FIG. 5 , primary storage component 210 has currentgeneration number 212 equal to G0+5 (meaning next IO will get gen G0+6)and has expected generation number 214 equal to G0+1 (meaning oldest IOrequest completed locally but not remotely, i.e., still unacknowledged,has generation number G0+1). Owner 210 also has bitmap 216 showing IOrequest G0+2 as having completed locally and remotely. On backup storagecomponent 220, current generation number 222 is G0+4 (which is thelargest generation number backup storage component 220 has seen) and hasexpected generation number 224 equal to G0+1 (which is the lowest unseengeneration number). Backup bitmap 226 shows IO requests G0+2 and G0+4 ashaving completed.

Primary storage component 210 can recognize a failure of backup storagecomponent 220 if backup storage component 220 stops acknowledgingreplications of IO requests, e.g., if the current generation number 212exceeds expected generation number 214 by a threshold difference, or ifprimary storage component 210 encounters errors when sending requests tobackup storage component 220. Other mechanisms (such as clustermembership protocols in a storage platform including multiple storagecomponents) can detect failure of backup storage component 220 andinform primary storage component 210 of the failure. If backup storagecomponent 220 fails, primary storage component 210 may stop sendingcopies of IO requests to backup storage component 220, e.g., may skipprocess block 314 in process 310 of FIG. 3-1 , and may process IOrequests locally only and keep incrementing values of current andexpected generation numbers 212 and 214.

If the primary storage component 210 fails, backup storage component 220may complete any pending backup IO requests and assume the role of theprimary storage component for volume 122. In general, storage componentsin the storage platform may use cluster membership protocols to detectand report on the failure of any member. Upon detection of failure ofprimary storage component 210, backup storage component 220 may initiatea process for taking over ownership of volume 122. For example, ifprimary storage component 210 fails, data stored for backup volume 124may be made accessible as the data of primary volume 122, and thestorage platform recognizes storage component 220 as owner of theprimary volume 122. Failed storage component 210 may then be considereda failed backup storage component.

A failed backup storage component, when returning to service, maycontact the primary storage component of the backed-up volume toreestablish redundancy. In particular, each storage component may keepits expected generation number in persistent or non-volatile memory, sothat when the other storage component fails and returns, the value ofthe expected generation number that the returning storage component hasfor the backed-up volume is still available, and the returning storagecomponent may report its last valid expected generation number to thecurrent owner of the volume. At that time, the owner may pause IO andsend its current and expected generation number values to the restored(now backup) storage component and also the generation numbersassociated with any snapshots that were created between current and thelast valid expected generation numbers of the backup storage component.The backup storage component may then set its current and expectedgeneration numbers to match the owner's values. The backup storagecomponent may also create snapshots with the given generation numbers.When the backup storage component informs the owner that the values ofits current and expected generation numbers match the values of thecurrent and expect generation number that the owner has, the ownerresumes IO and starts replicating all further IO requests. The backupstorage component then starts a background resynchronization process,whereby the backup storage component asks the owner to send any data theowner has applied between the last valid expected generation number. Theowner can send such data in multiple batches, and the backup storagecomponent may apply such data to its backend storage. When the backupstorage component has fully caught up with IO requests that the ownerprocessed while the backup storage component was unavailable, the ownerand backup storage component declare the volume has reestablishedredundancy.

The state of storage platform 200 shown in FIG. 5 is an example of astate that platform 200 may have when backup storage component 220fails, and primary storage component 210 declares backup storagecomponent 220 dead. When primary storage component 210 declares backupstorage component 220 dead, primary storage component 210 saves expectedgeneration number 214 in persistent storage. In the example state ofFIG. 5 , that will be G0+1. From the state shown in FIG. 5 , primarystorage component 210 completes IO requests G0+4 and G0+5 locally andcontinues to operate normally except that primary storage component 210stops forwarding further IO transfers to backup storage component 220.Current and expected generation number 212 and 214 keep increasing asprimary storage component 210 continues to handle IO transfers to theowned volume 122. Backup storage component 220, which failed, has valueG0+1 for last valid expected generation number 224 saved in itspersistent memory.

A backup storage component when returning to service after a failure maycommunicate with the primary storage component to resume backup service.FIG. 6 is a flow diagram of a process 600 for a primary storagecomponent and a backup storage component to resynchronizes a backupvolume with a primary volume. To provide an illustrative example,process 600 is described in the following with reference to storageplatform 200 of FIG. 2 in the case where backup storage component 220failed when platform 200 was in the state illustrated in FIG. 5 . In aprocess block 610, backup storage component 220 retrieves the value ofexpected generation number 224 from persistent storage and informsprimary storage component 210 of the last valid expected generationnumber 224, which in the example of FIG. 5 was G0+1.

Backup storage component 220 in a process block 620 optionally removesall IO requests that had generation numbers greater than or equal toG0+1. For example, backup storage component 220 may use bitmap 226 ormay query data index 227 to find IO requests having generation numbersG0+2 and G0+4 were applied. In performing block 620, backup storagecomponent 220 removes all entries in data index 227 having a keycontaining generation numbers greater than the expected generationnumber, e.g., entries corresponding to generation numbers G0+2 and G0+4,and performs any cleanup processes to restore the backup volume 124 to astate before the IO request having the last expected generation numberG0+1.

Backup storage component 220, in a process block 630, contacts primarystorage component 210 after the cleanup (if process block wasperformed). At that time, owner 210 generally has a current generationnumber 212 that is greater than expected generation number 224. Forexample, current generation number 212 may be G0+200. Primary storagecomponent 210, in process block 630, temporarily blocks IO requestschanging volume 122 and tells backup storage component 220 to setgeneration numbers 222 and 224 to G0+200 and G0+201. IO to volume 122may be blocked for fractions of a second, e.g., the time needed forprimary storage component 210 to block the IO requests, send thecurrent/expected generation numbers, and re-enable replication to ensureany new IO requests will be forwarded to backup storage component 220,before primary storage component 210 unblocks IO requests. Primarystorage component 210 and backup storage component 220 then have thesame current and expected generation numbers, and primary storagecomponent 210, in a process block 640, may resume replicating new IOrequests and sending the new IO requests to backup storage component220. At this point, backup storage component 220 has not applied IOrequests G0+1 to G0+200 to backup volume 124, and in a process block650, primary storage component 210 can reproduce the missing IO requestsand send the reproduced IO request to backup storage component 220 forapplication to backup volume 124.

Process block 650 may, for example, include backup storage component 220asking primary storage component 210 to send backup storage component220 write data belonging to volume 122 and having generation numbers inthe range of G0+1 to G0+200 inclusive. Data index 217 has entriesindexed by generation numbers of IO requests and mapping to physicalstorage locations were the data corresponding to the IO request isstored in backend media. Primary storage component 210 can thusreconstruct the IO requests having generation numbers from G0+1 toG0+200 that are still relevant to data in volume 122 and therefore tobackup volume 124. (An IO request may not be relevant, for example, ifdata for the IO request was overwritten in response to a later IOrequest.) The reproduced IO requests may be sent with a flag or othermetadata that distinguishes the reproduced missing IO request from a newIO request. Once all data with generation numbers between G0+1 andG0+200 has been sent to backup storage component 220 from primarystorage component 210 and have been applied in backup storage component220, primary storage component 210 and backup storage component 220 areagain in sync, i.e., have been resynchronized.

A resynchronization process can avoid process block 620 (in which thebackup storage component 220 deletes IO requests with generation numbergreater than or equal to its last valid expected generation numberbefore failure) by checking reconstructed IO requests duringresynchronization process to determine whether the IO request wasalready applied. For example, when backup storage receives thereproduced IO requests, backup storage component 220 can check to bitmap226 or data index 227 and persistent storage 252 to determine whether anidentical IO requests was already applied. If so, backup storagecomponent 220 can acknowledge that reproduced IO request withoutapplying it. Alternatively, the backup storage component 220 in processblock 640 may ask primary storage component 210 to send backup storagecomponent 220 only the IO requests that backup storage component 220missed, e.g., G0+1, G0+3, and G0+5 to G0+200.

Volume 122 may have snapshots, when backup storage component 220 returnsto service and contacts primary storage component 210. In this case,backup storage component 220 has in its persistent store last validexpected generation number 224 and tells primary storage component 210this value (in process block 610). Primary storage component 210 pausesIO and tells backup storage component 220 the values of current andexpected generation numbers 212 and 214 (in process block 630). Primarystorage component 210 also tells backup storage component 220 anygeneration numbers associated with snapshots between current generationnumber 212 and the last valid expected generation number of backupstorage component 220. Backup storage component 220 sets current andexpected generation numbers 222 and 224 to match the owner's values.Backup storage component 220 may then also create each snapshot withmatching generation numbers corresponding to the snapshot.

A resynchronization or synchronization process can be used if a newstorage component is designated for backup. For example, if an SPU orother storage component is newly assigned to be the backup storagecomponent 220 that backups an existing volume 122, the backup storagecomponent 220 needs to go through a full synchronization process. Forthat, backup storage component 220 tells primary storage component 210that backup storage component 220 is ready to receive replicated data.Primary storage component 210 blocks 10 and sets backup storagecomponent 220 to have current and expected generation numbers 222 and224 according to the value of current generation number 212 of volume122. Primary storage component 210 unblocks 10 and starts replicatingthe modification requests targeted at volume 122.

If the backup storage component 220 dies again before aresynchronization process has completed. Backup storage component 220 isnot in a synced state. Backup storage component 220 is still in aresynchronizing state. So backup storage component 220 does not (and didnot) overwrite the last valid expected generation number in thepersistent store of backup storage component 220. In the example of afailure in the state shown in FIG. 5 and subsequent failure beforeresynchronization was achieved, backup storage component 220 still hasG0+1 as expected generation number in its persistent resynchronizationinformation. When backup storage component 220 again comes back up,backup storage component 220 starts getting all data from G0+1 towhatever value current generation number 212 on primary storagecomponent 210 has at that time. This may cause some data that wasalready sent to the backup storage component 220 to be sent again.Backup storage component 220 can use data index 227 to recognize theduplication and throw away duplicate data.

If owner 210 and backup 220 are in the state shown in FIG. 5 whenprimary storage component 210 fails, primary storage component 210remembers has the value (G0+1) of its expected generation number 214 inpersistent memory. Backup storage component 220 takes over IO requeststo volume 122 using the data of backup volume 124. At that time, basedon its value (G0+4) of current generation number 222, backup storagecomponent 220 sets expected generation number 224 to one more thancurrent generation number 222 (G0+5), but storage component 220 canfurther apply all forward IO requests that where in flight when storagecomponent 210 failed. The new owner 220 is now handling IO locally onlyand not replicating it, so that current generation number 222 may keepgoing up. When the failed storage component 210 comes back up,resynchronization process 600 as described above is performed withcomponent 220 as the owner and component 210 as the backup.

Each of modules disclosed herein may include, for example, hardwaredevices including electronic circuitry for implementing thefunctionality described herein. In addition or as an alternative, eachmodule may be partly or fully implemented by a processor executinginstructions encoded on a machine-readable storage medium.

All or portions of some of the above-described systems and methods canbe implemented in a computer-readable media, e.g., a non-transientmedia, such as an optical or magnetic disk, a memory card, or othersolid state storage containing instructions that a computing device canexecute to perform specific processes that are described herein. Suchmedia may further be or be contained in a server or other deviceconnected to a network such as the Internet that provides for thedownloading of data and executable instructions.

Although particular implementations have been disclosed, theseimplementations are only examples and should not be taken aslimitations. Various adaptations and combinations of features of theimplementations disclosed are within the scope of the following claims.

What is claimed is:
 1. A process comprising: for a sequence of requeststo write data in a virtual volume owned by a first storage processingunit, the first storage processing unit processing each of the requestsby: assigning to the request to a generation number sequentially from asequence of generation numbers; applying the request to first persistentstorage associated with the first storage processing unit; andtransmitting the request and the generation number of the request to asecond storage processing unit; for each of the requests the secondstorage processing unit receives from the first storage processing unit,the second storage processing unit processing the request by: applyingthe request to second persistent storage associated with the secondstorage processing unit; and updating resynchronization data of thesecond storage processing unit, the resynchronization datadistinguishing which generation numbers from the sequence of generationnumbers do not correspond to the requests processed by the secondstorage processing unit.
 2. The process of claim 1, wherein in responseto the second storage processing unit returning to service after beingunavailable, the process further comprises: the second storageprocessing unit retrieving the resynchronization data and informing thefirst storage processing unit of a set of generation numbers for whichthe second storage processing unit has not process requests; and thefirst storage processing unit creating reconstructed requestscorresponding to generation numbers in the set, and transmitting thereconstructed requests to the second storage processing unit.
 3. Theprocess of claim 1, wherein for each of the requests that the secondstorage processing unit receives from the first storage processing unit,the second storage processing unit further processes the request bysending an acknowledgement to the first storage processing unit, theacknowledgement indicating that the second processing unit has appliedthe request.
 4. The process of claim 3, further comprising: the firstprocessing unit maintaining a current generation number indicating thegeneration number last assigned to one of the requests; and the firstprocessing unit updating an expected generation number in response toreceiving from the second processing unit an acknowledgementcorresponding to a prior value of the expected generation number.
 5. Theprocess of claim 1, wherein the second processing unit updating theresynchronization data, comprises: the second processing unit, inresponse to the generation number of the request being greater than avalue of a current generation number kept by the second processing unit,setting the current generation number to the generation number of therequest; and the second processing unit, in response to the generationnumber of the request matching a value of an expected generation numberkept by the second processing unit, changing the value of the expectedgeneration number to a generation number that follows in the sequence ofgeneration numbers and does not correspond to any of the IO requestsprocessed by the second processing unit.
 6. The process of claim 1,wherein the resynchronization data comprises: a current generationnumber indicating a greatest of the generation numbers of the requeststhat the second storage processing unit has applied; and an expectedgeneration number indicating a least of the generation number of therequest that the second storage processing unit has not applied.
 7. Theprocess of claim 6, wherein the resynchronization data further comprisesa bit map containing bits that map to generation numbers and are set ornot depending on whether the second storage processing unit hasprocessed the request corresponding to the generation numbers.
 8. Astorage processing unit comprising: a persistent memory; and a processorconfigured to process a sequence of requests to write data in a virtualvolume associated with the storage processing unit, the storageprocessing unit processing each of the requests by: assigning to therequest to a generation number sequentially from a sequence ofgeneration numbers; applying the request to first persistent storageassociated with the first storage processing unit; transmitting therequest and the generation number of the request to a backup storagecomponent that maintains a backup of the virtual volume; maintaining inthe persistent memory a current generation number indicating thegeneration number last assigned to one of the requests; and in responseto receiving from the backup storage component an acknowledgementcorresponding to a value of an expected generation number in persistentmemory, updating in the expected generation number.
 9. The storageprocessing unit of claim 8, wherein the processor is further configuredto execute a resynchronization process after recovering from a failedstate, the resynchronization process comprising: accessing values of thecurrent generation number and the expected generation number from thepersistent memory; communicating the values of the current generationnumber and the expected generation number to the backup storagecomponent; receiving from the backup storage component a set ofreconstructed requests that correspond to generation numbers determinedbased on the values of the current generation number and the expectedgeneration number from the persistent memory.
 10. The storage processingunit of claim 8, wherein the processor is further configured to executea resynchronization process after the backup storage component recoversfrom a failed state, the resynchronization process comprising: receivingfrom the backup storage component resynchronization data indicatinggeneration numbers corresponding to requests that the backup storagecomponent has not applied; reconstructing a set of reconstructedrequests that the storage processing unit has applied and thatcorrespond to generation numbers of the requests that the backup storagecomponent has not applied; and transmitting the reconstructed request tothe backup storage component.
 11. A storage processing unit comprising:a persistent memory; and a processor configured to process a sequence ofrequests to write data in a virtual volume associated with a primarystorage component, the storage processing unit processing each of therequests by: receiving the request and a generation number of therequest from the primary storage component, the generation number beingfrom a sequence of generation numbers; applying the request topersistent storage associated with the storage processing unit;acknowledging to the primary storage component that the storageprocessing unit applied the request; and updating resynchronization datain the persistent memory, the resynchronization data distinguishingwhich generation numbers from the sequence of generation numbers do notcorrespond to requests processed by the storage processing unit.
 12. Thestorage processing unit of claim 10, wherein updating theresynchronization data comprises: in response to the generation numberof the request being greater than a value of a current generation numberin the persistent memory, setting the current generation number to thegeneration number of the request; and in response to the generationnumber of the request matching a value of an expected generation numberin the persistent memory, changing the value of the expected generationnumber to a generation number that follows in the sequence of generationnumbers and does not correspond to any of the TO requests processed bythe second processing unit.
 13. The storage processing unit of claim 12,wherein the processor is further configured to execute aresynchronization process after recovering from a failed state, theresynchronization process comprising: accessing values of the currentgeneration number and the expected generation number from the persistentmemory; communicating the values of the current generation number andthe expected generation number to the second storage processing unit;receiving from the second storage processing unit a set of reconstructedrequests that correspond to generation numbers determined based on thevalues of the current generation number and the expected generationnumber from the persistent memory.